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Abstract 



Several high-level committees have concluded that current domestic and global trends are 
threatening America’s global science and engineering (S&E) preeminence. Of the challenges 
discussed, few are thought to be as serious as the purported decline in the supply of high quality 
students from the beginning to the end of the S&E pipeline — a decline brought about by 
declining emphasis on math and science education, coupled with a supposed declining interest 
among domestic students in S&E careers. 

However, our review of the data fails to find support for those presumptions. Rather, the 
available data indicate increases in the absolute numbers of secondary school graduates and 
increases in their math and science performance levels. Domestic and international trends 
suggest that that U.S. schools show steady improvement in math and science, the U.S. is not at 
any particular disadvantage compared with most nations, and the supply of S&E-qualified 
graduates is large and ranks among the best internationally. Eurther, the number of 
undergraduates completing S&E studies has grown, and the number of S&E graduates remains 
high by historical standards. Why, then, is there a purported failure to meet the demand for S&E 
college students and S&E workers? 

Analysis of the flow of students up through the S&E pipeline, when it reaches the labor market, 
suggests the education system produces qualified graduates far in excess of demand: S&E 
occupations make up only about one-twentieth of all workers, and each year there are more than 
three times as many S&E four-year college graduates as S&E job openings. So it is not clear, 
even if there were deficiencies in students’ average S&E performance, that such deficiencies 
would necessarily be insufficient to meet the requisite S&E demand. While improving average 
math and science education at the K-12 level may be warranted for other reasons, such a strategy 
may not be the most efficient means of supplying the S&E workforce. 

Workforce development and education policy requires a more thorough analysis than appears to 
be guiding current policy reports. The available evidence points, first, to a need for targeted 
education policy, to focus on the populations in the lower portion of the performance 
distribution. Second, the seemingly more-than-adequate supply of qualified college graduates 
suggests a need for better understanding why the “demand side” fails to induce more graduates 
into the S&E workforce. Third, public and private investment should be balanced between 
domestic development of S&E workforce supply and global collaboration as a longer-term goal. 
Policy approaches to human capital development and employment from prior eras do not address 
the current workforce or economic policy needs. 
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Into the Eye of the Storm: 

Assessing the Evidence on Science and Engineering 
Education, Quality, and Workforce Demand 



Policymakers and industry leaders are once again concerned about the adequacy of the science 
and engineering (S&E) workforce. A growing number of reports claim that a lack of sufficient 
numbers of scientists and engineers entering the workforce is threatening the United States’ 
economic health and dominant position in global innovation. The primary causes of an 
impending workforce shortage, it is argued, are the mediocre preparation of domestic students in 
the educational pipeline and an ongoing decline in their interest in pursuing S&E careers. To 
address the assumed crisis, the consensus recommendation of business groups, public 
policymakers, and educators is to expand and improve science and math education from 
kindergarten through college, and to more aggressively court foreign S&E students and workers. 

This paper examines the assumptions about the state of the educational pipeline and the 
purported workforce shortages. Despite this nearly universal support for upgrading science and 
math education, our review of the data leads us to conclude that, while the educational pipeline 
would benefit from improvements, it is not as dysfunctional as believed. Today’s American high 
school students actually test as well or better than students two decades ago. Eurther, today’s 
students take more science and math classes, and a large number of students with strong science 
and math backgrounds graduate from U.S. high schools and start college in S&E fields of study. 
Graduate schools have an ample pool of qualified four-year graduates to draw from but seem 
unable or unwilling to do so. Surprisingly few of the many students who start along the path 
toward S&E careers take the next steps to remain in an S&E career. If there is a problem, it is not 
one of too few S&E qualified college graduates but, rather, the inability of S&E firms to attract 
qualified graduates. The pool of graduates with an S&E degree exceeds the number of S&E job 
openings each year, even though employers may not be as successful as they would like in 
attracting or retaining graduates into an S&E career. 

The various policy reports focusing on increasing the science and math preparation at the K-12 
level almost uniformly fail to ask the question our analysis suggests — has the increase in the 
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absolute numbers of secondary school graduates and the increase in their math and science 
performance levels provided an adequate number of domestic S&E college majors? 

The pool of S&E-qualified secondary and postsecondary graduates is several times larger than 
the number of annual job openings. The flow of secondary school students up through the S&E 
pipeline, when it reaches the labor market, supplies occupations that make up only about a 
twentieth of all workers. So even if there were deficiencies in students’ average science and 
math performance, such deficiencies would not necessarily deplete the requisite supply of S&E 
college majors. Even if modal test scores or course-taking was by some measure low, the size of 
the graduating student body is so large that there would still be a sufficient number of students 
who test above average and who are fully qualified for the relatively small number of S&E jobs. 
While improving average math and science education at the K-12 level may be warranted for 
other reasons, such a strategy may not be the most efficient means of supplying the S&E 
workforce. 

Our analysis at the aggregate level does not find a shortage of potential S&E students or 
workers. However, this conclusion is put forth with one caveat: the analysis of all S&E students 
and workers may not apply equally to the trends and problems faced in specific fields or by 
domestic minority groups. A fine-grained analysis of specific industries, occupations, and 
populations is needed to identify the weakness in the U.S. education system. We are, indeed, 
conducting this level of analysis for future reports. The S&E world includes a broad range of 
knowledge, types of related jobs, and a great diversity of students and workers with academic 
performance and employment trends different from the overall averages. A better understanding 
of S&E workforce demand and education and workforce development will identify areas where 
public and private policy could be most effectively targeted. 

This paper begins with a look at U.S. high school students and the data on domestic S&E course- 
taking and tests of performance, and then it examines how U.S. students compare on 
international tests of S&E performance. Our intent is to trace students along the pipeline or the 
“pathways” from school to work. In the next section of the paper, we examine data on reported 
interest in an S&E studies by college entrants and the subsequent graduation of S&E 
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degreeholders. The following section examines the relative size of the graduating class, as well 
as the relative size of S&E employment growth. There are transition points along S&E pathways 
and we wish to estimate the proportion of students at each transition point who continue on to the 
next S&E point in the pathway. Eor example, where does the purported shortfall of native S&E 
supply take place: at high school, college, or at entry into the labor market? How much has the 
proportion of S&E graduates pursing an S&E career changed? In the final section of the paper, 
we note that employer reports of hiring problems should not be taken as prima fascia evidence of 
labor market shortages. Moreover, education and immigration policies that are based on that 
assumption can undermine future economic and labor-market strength. We consider these often 
overlooked, negative long-term consequences of implementing well-meaning but misguided 
policies. 

Concerns with the Educational Pipeline 



The conclusions several high-level committees have reached within the past few years reinforce 
the perception that domestic and global trends challenge America’s global S&E preeminence. Of 
the various challenges discussed, few are thought to be as serious as the purported decline in the 
supply of high-quality students from the beginning to the end of the S&E pipeline. It is 
concluded that the combination of the educational weaknesses in domestic supply, along with 
growing S&E capacity abroad, has reached a dangerous and threatening stage: 



The future well-being of our nation and people depends not just on how well we educate 
our children generally, but on how well we educate them in mathematics and science 
specifically.. . . Our children are falling behind; they are simply not “world-class learners” 
when it comes to mathematics and science. (The National Commission on Mathematics 
and Science Teaching for the 21st Century 2000) 

If trends in U.S. research and education continue, our nation will squander its economic 
leadership, and the result will be a lower standard of living for the American people. . . . 

By 2015 [the country needs to] double the number of bachelor’s degrees awarded 
annually to U.S. students in science, math, and engineering. (National Summit on 
Competitiveness 2005) 

The United States faces an unprecedented challenge to its long-term global economic 
leadership. And a fall from leadership would threaten the security of the nation and the 
prosperity of its citizens.. . . High school students in the U.S. perform well below those in 
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other industrialized nations in the fields of mathematics and science . . . [and thus we need 
to make] STEM education a national priority. (Council on Competitiveness 2004) 

The committee is deeply concerned that the scientific and technological building blocks 
critical to our economic leadership are eroding at a time when many other nations are 
gathering strength ... we fear the abruptness with which a lead in science and technology 
can be lost. ... (National Academies of Science 2006) 

Reviewing the various reports, and in light of the data on science and math preparation, we 
identify three major concerns in the reports on weaknesses in U.S. students’ science, math, and 
technology preparation. First, it is argued that the low levels of skills and math and science 
proficiency of U.S. students lead to an inadequate supply of college students and college 
graduates for the science and engineering workforces. 

Second, it is further argued that U.S. students’ math and science proficiencies are lower than 
those of students around the globe. Other nations are improving and expanding their science, 
math, and engineering education at higher rates than the United States, putting the United States 
at a competitive disadvantage. The reports forcefully argue that addressing an educational deficit 
of U.S. high school students is necessary to maintain a competitive advantage. Presumably, the 
fact that high school students elsewhere perform better also translates into, or at least is 
correlated with, the growing production of S&E workers in the rising economic powers China 
and India. ^ 



Third, it is argued that any improvement in the math and science achievement of U.S. students is 
not occurring fast enough. Internationally, the rapid expansion of education systems in the 
emerging economies is presumed to be producing students with achievement levels improving at 



* The proportion of immigrants, who presumably come from countries with stronger science education, has grown in 
U.S. colleges and in the S&E workforce. These increases are taken to mean that demand has outstripped the supply 
of adequately trained domestic students. For example, the Business Roundtable concludes that 

in the face of the declining interest and proficiency by Americans in science, math and engineering, 

American industry has become increasingly dependent — some would say overly dependent — on 
foreign nationals to fill the demand for talent in a variety of fields that require strong backgrounds in 
science, technology, engineering and mathematics. (Business Roundtable 2003) 

We will not address this latter concern in depth, but suffice it to say that the competition for S&E students is 
growing worldwide, the potential supply from abroad remains strong, and it is unclear that the United States must 
retain the greatest share of the global student body to remain competitive. More to the point, the United States will 
retain the lion’s share of the global student body under almost any future scenario, and it is unclear that a race to 
retain a numerical majority will ensure that the United States retains the best and the brightest students. 
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a rate outpacing that of U.S. students. Domestically, many argue that improvements are not 
occurring fast enough to close the achievement gaps between different demographic groups. This 
means primarily that many women, Latinos, and African Americans are disadvantaged in 
competing for good jobs. 

In combination, these presumed deficiencies in U.S. education are viewed as major shortcomings 
in the future “competitiveness,” innovation, and economic performance capabilities of the United 
States. In the following sections, we review the evidence on educational quality and then analyze 
the actual supply of science, math, and engineering students and graduates. 

High School Performance: Bad and Getting Worse? 

We begin by examining the evidence supporting the common assertions that the United States 
has long been failing in S&E education and that its students perform abysmally when compared 
to international norms. These assertions rest heavily on several widely cited “facts” that are at the 
least ambiguous and, when examined a little more closely, appear incorrect. The reports cite 
well-known national and international data sources purported to demonstrate three deficiencies 
in our system: (a) inadequate and declining math and science preparation: the number of 
courses taken are few and test scores have been declining over the past 20 to 30 years; (bj poor 
performance on international tests: U.S. students have lower levels of math and science 
knowledge when compared to those in many other countries; and (c) the inferior structure and 
content of U.S. education: the overall K-12 educational curriculum and the preparation of its 
teachers are of lower quality than in other countries. 

Science and Math Coursetaking and Test Performance 

The first assertion in the reports is that education levels have been in decline overall and that 
U.S. students are not doing well in math and science in particular. The supporting data, when 
provided, are of the percentages of low-performing students supplemented with employer 
testimony about poorly prepared applicants. Although the proportion of poorly performing 
students is an important concern about the education system, and one we address later in this 
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paper, the relevant indicator of science and engineering supply is the performance and size of the 
population that is qualified for the science and engineering workforce. 

The proportion of students who at least finish a high school degree has increased notably over 
the past 30 years. There is some debate over the precise rate of completion and the appropriate 
measure to use, but the most widely accepted “status completion” rate for 18- to 24-year-olds 
compiled by the National Center for Educational Statistics was 83 percent in 1972 and increased 
to 87 percent by 2004. The different methods result in completion rates that vary, but the trends 
are similar, showing significant increase in completion rates over the past 20 to 30 years, with 
slow to marginal increases more recently. Although the high school completion rate is lower for 
certain groups, there has been steady improvement in high school completion for every 
demographic group. At the same time, more students are staying in school and more of the 
student population is “on track,” defined as enrolled at the modal grade level for their age. 
Between 1994 and 2003, there was a 6 percentage point increase to 75 percent of 12- to 17-year- 
olds who were academically “on track” (Dye and Johnson 2006). 



High school students’ exposure to science and math has increased over time. In 1982, high 
school graduates earned 2.6 math credits and 2.2 science credits on average. By 1998, the 
average number of credits increased to 3.5 math and 3.2 science credits. Further, the share of 
students who take algebra early increased from 1986 to 1999. The percentage of 13-year-olds 
enrolled in algebra and in prealgebra rose 38 and 78 percent, to 22 and 34 percent from 16 and 
19 percent, respectively (NCES 2001a). Students from all racial/ethnic groups, and both male 
and female students, significantly increased science and math course-taking, albeit differential 
achievement rates between groups remain. Table 1 shows the trends from 1990 to 2000. 



^ Cited here are the National Center for Education Statistics (NCES) data that are from the annual dropout report 
based on self-reported high school completion in the Current Population Survey. White non-Hispanics complete at 
92 percent and blacks at 83 percent. Today, the lowest rate is for Hispanic immigrants with a completion rate of 55 
percent and the next lowest is for native-born Hispanics at 81 percent. A more narrowly defined status completion 
rate for 17- to 19-year-olds reduces the impact of nativity and improves the completion rates of foreign-born adults 
(because so many non-high school completers immigrate in their early twenties, increased rates of immigrants with 
less than a high school education may lead to lower graduation rate percentages in some estimates that use total 
post-high school population). These data include GED degrees which may make up as much as a seventh of those 
who report high school completion, and the data on GED are not reliable. Nevertheless, we prefer, along with most 
analysts, these traditional indicators of high school completion, but we are aware of lower rates based on a measure 
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Table 1. Average number of Carnegie units earned by 
public high school graduates 





Total 


Math 


Science 


Foreign 

language 


1990 graduates 


23.5 


3.2 


2.8 


1.5 


2000 graduates 


26.1 


3.6 


3.2 


2 



Source: NCES 2001a 



Further, the Couneil of Chief State Sehool Officers report that the majority of states now require 
three or four years of high school mathematics and two or three years of high school science 
(Blank and Langesen 2005). There have been significant gains in course-taking at the national 
level, increasing from only 45 percent of students taking chemistry in 1990, to 55 percent in 
1996, and 60 percent in 2004. The proportion taking three years of math increased from 49 
percent in 1990 to 72 percent in 2004, and the proportion of students completing four years of 
math increased from just 29 percent in 1990 to 37 percent in 1994, and to 50 percent in 2004. 
Similarly, the number of math and science qualified instructors has increased notably. Of course, 
the distribution of improvement matters, and there are substantial differences between states and 
regions and between different demographic groups. Similar trends in math and science are 
evident among college-bound seniors taking the SAT, though a drop in English composition also 
occurred (see table 2). 



Table 2. How Have College-Bound Seniors Changed in 10 Years? 



Based on college-bound seniors who took the SAT Reasoning Test prior to high school 
graduation In 1997, 2006, and 2007. 

1997 2006 2007 



English composition 


71% 


62% 


66% 


Precalculus 


40 


50 


53 


Calculus 


23 


28 


30 


Physics 


48 


53 


54 


Chemistry 


85 


88 


89 


U.S. government/civics 


70 


67 


70 


Economics 


48 


45 


48 


Spanish 


65 


70 


70 



Source: 2007 College-Bound Seniors: Total Group Profile Report. College Board 



of high school graduates divided by the 17-year-old population. Longitudinal data on individuals confirm increasing 
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NAEP Math Scores 




□ 13 year olds □ 17-year-olds 



Figure 1 



At the same time, the math and science performance of high school graduates is not declining 
and shows improvement for some grades and demographic groups, especially over the long run. 
The data in figure 1 from the National Assessment of Educational Progress (NAEP) show 
improvements in math test scores: 13-year-old cohorts show steady improvement from 1973 to 
2004, while 17-year-old cohort performance fluctuated but still tested better in math in 2004 than 
30 years previously (NCES 2001a). The NAEP science scores, which were collected starting 
in 1996, show no significant changes for 4th and 8th graders by 2000. There was a slight decline 
in science scores among 12th graders from 1996 to 2000 (NCES 2002). Although the 17-year- 
old and 12th grade test results may indicate less improvement in educational quality at the high 
school level, the scores may also reflect changes in cohort mix over time resulting from 
increased high school retention. It would be difficult to separate performance change due to 
education quality and changes in cohort mix if the cohort tested now includes more lower- 
performing students who, in the past, would have dropped out of school and thus not been tested. 
The past few decades have witnessed a narrowing of the achievement gaps between men and 
women and between race/ethnic groups. In some math and science areas, women now have 



graduation rates over the past three decades (NCES 2006). 
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higher achievement levels, take more courses, and enter some S&E fields in greater numbers 
than men (NCES 2001a). ^ 



The well known Scholastic Aptitude Tests (SAT) also show an increase in scores over the past 
three decades (College Board 2006). Eigure 2 shows, as SAT followers know, a dip in math 
scores occurred during the 1970s through the mid-1980s, but there has been a steady increase 
since that time. The SAT math scores increased sharply between 1996 and 2006 from a score of 
508 to 518 (adjusted for the break in standardization during the period). Eurthermore, the gap in 
female and male math scores has closed from about 40 points at the outset of the 1980s to 30 
points today. In the past year, there has been a slight, though statistically insignificant, decline of 
1 to 3 points. However, there were also more students and more diverse groups of students taking 
this test, which, it is suggested, is the likely cause of the decline; compared with the past, test 
takers are more diverse in terms of demographics and academic performance (e.g., in some 
states, all high school seniors were required to take the test)."^ As the president of the College 



^ For detail on math and science course taking and achievement over the past 20 to 30 years, NSF provides the most 
comprehensive review. The following summary of the research comes from the NSF Science and Engineering 
Indicators 2004: “The NAEP trend assessment shows that student performance in mathematics improved overall 
from 1973 to 1999 for 9-, 13-, and 17-year-olds.... In general, the average performance of both males and females in 
mathematics improved from the early 1970s to the late 1990s, including the period from 1990 to 1999. In 1982, high 
school graduates earned an average of 2.6 mathematics credits and 2.2 science credits. By 1998, those numbers grew 
to 3.5 and 3.2 credits, respectively (NCES 2001a). This expansion of academic course taking included all 
raciaFethnic groups and both male and female students.... NAEP data indicate that the proportion of students who 
take algebra early increased between 1986 and 1999 ... [the percentage of] 13 year olds enrolled in algebra and in 
prealgebra... had risen [from 16 and 19 percent] to 22 and 34 percent, respectively [increases of 38 and 78 percent, 
respectively]. 

“The percentage of high school graduates who enrolled in postsecondary education immediately after graduation has 
increased over the past 3 decades, rising from 47 percent in 1973 to 62 percent in 2001 (NCES 2003a).. . . Between 
1973 and 2001, the rate at which females enrolled in postsecondary institutions increased from 43 to 64 percent, 
whereas the rate for males increased from 50 to 60 percent. The immediate enrollment rate for white high school 
graduates increased from 48 percent in 1973 to 64 percent in 2001. Eor black graduates, the immediate enrollment 
rate increased from 32 percent in 1973 to 55 percent in 2001 . . . the gap between the two groups [blacks and whites] 
has diminished since 1983 [but] the gap between Hispanic students and white students has increased. The gap in 
immediate postsecondary enrollment rates between high school graduates from high- and low-income families 
persisted from 1973 to 2001. 

“By 2000, women earned half the [S&E] degrees, up from one-third [in mid-1970s]. Degrees awarded to 
underrepresented minorities rose from 9 to 16 percent, and those awarded to Asian/Pacific Islanders increased from 
2 to 9 percent.” 

The test taking group now includes more students who traditionally have lower academic performance. 

Eor example; “In all, 35 percent of the class taking the SAT would be the first in their family to attend college, 
mirroring the broad diversification of the college student population that university officials have been noting and 
encouraging in recent years. . . Eor example, the average score for students who planned to apply for financial aid in 
college was 501 in critical reading and 508 in math; the average scores for students who did not intend to apply for 
aid was 530 in critical reading and 548 in math. The average scores for students whose parents did not graduate from 
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Board, which administers the SAT, explained, “The larger the population you get that takes the 
exam, it obviously knocks down the scores” (Finder 2007; College Board 2007). 

The other college entrance exam, the ACT, which is taken by only a slightly smaller 
number of students (in 2007, 1.3 mi llion took the ACT, compared with 1.5 million who took the 
SAT), shows steadily increasing achievement levels among students through the late 1990s, a 
decline between 2002 and 2005, and then a steep increase in 2006 and continuing to reach a ten- 
year high in 2007. This occurred with a steady increase in the numbers of students taking the 
ACT over this period. 




Additionally, one international test provides additional evidence that there was no decline in U.S. 
student aehievement over the 1990s. The TIMSS (Trends in International Mathematics and 
Science Study), whieh is administered to third and eighth graders in up to 46 countries, is 
frequently cited as evidence of poor performance by U.S. students relative to students in other 
nations (we return to this issue below). Yet, the TIMSS, which has been administered to third 
and eighth graders in 1995, 1999, and 2003 (with a 2006/2007 wave not yet reported) also finds 
that, looking just at the scores of U.S. students, there has been no decline and even some 
improvement over this period (Gonzales 2004). Across all U.S. race, gender, and income groups, 
there was either no ehange or there was improvement over the three waves of the TIMSS for 
eighth graders and small variations for fourth graders. In many eases, gaps between race and 

high school was 421 in critical reading and 445 in math; the comparable averages for students whose parents are 



10 



gender groups narrowed. No consistent, notable declines were found in detailed analyses of the 
survey results. 

The weight of the evidence, when considering all the different measures, surely indicates no 
decline but rather indicates an ongoing educational improvement for U.S. students. This 
improvement is not only in math and science but in all subjects tested and, importantly, occurs at 
the same time as a greater and more diverse proportion of the population is remaining in school.^ 
But there is another arrow in the critics’ quiver: there is a global race afoot and the United States 
is losing pace so that it matters not if domestic trends are improving because U.S. students, 
relatively speaking, do not perform as well as students in other nations. We turn now to that 
issue. 

Strong Showings on International Tests 

The second claim is that U.S. economic and innovative capacity is imperiled by the relatively 
low performance of U.S. students compared with students in other countries. The test results 
cited as support of this claim range from the failure of the United States to be number one to an 
apparent decline from fourth grade rankings, which are relatively strong, to twelfth grade 
rankings, which are at or near the bottom of the list. However, although cross-national tests are 
valuable when used for carefully defined purposes, almost all of these tests do not support the 
conclusions often drawn about national “rankings” of student performance. 

Global Rankings and the TIMSS. The most widely cited study, noted above, is the TIMSS 
(Trends in International Mathematics and Science Study). The test has been administered to third 
and eighth graders and, in addition, in 1995 and 1999 the test was administered to twelfth 
graders. In the fourth and eighth grade math and science tests over the 1995 to 2003 period, the 
U.S. performs above the international average, with rank-orders between fifth and twelfth 
depending upon the year, the grade (fourth or eighth), and the test (math or science). The twelfth 



college graduates was 522 and 533.” (Finder, 2007) 

^ A reasonable inference is that lower performing and harder-to-serve students were disproportionately dropouts. 
Thus, achieving stable or improved high school test performance in a student population that includes more of these 
students, not formerly taking the tests as 17-year-olds, may understate the actual levels of improvement. 



11 




grade test administered just in 1995 and 1999 has been widely criticized as sampling very 
different populations in each country (in some cases including cohorts that cover a three year age 
range and disparate course taking) and neither the TIMSS nor other international tests currently 
test this age or grade group. 

The notion that the United States trails the world in educational performance misrepresents the 
actual test results and reaches conclusions that are quite unfounded. The test rankings and 
conclusions about the global standing of the United States and its education system typically fail 
to consider the membership of the global comparison group, the composition of the leading 
groups, the actual and meaningful differences in ranking, rankings on subjects other than math 
and science, American exceptionalism in its comparative heterogeneity, the extent of the “gap” 
between the groups, and the relationship between the tests and assumed outcomes of economic 
performance. 

Typically the list purporting to rank performance outcomes fails to note statistically significant 
differences and the extent of test score difference: many differences in scores are small and are 
not statistically significant in some cases. For example, there is no statistical difference between 
the “fifth ranked” United States in 2003 and the nations ranked third and fourth.^ Moreover, the 
list of countries participating does not include Brazil, China (mainland), Germany, France, India, 
or Mexico, among other economically important and populous nations. 

Although the United States does not lead the list in any particular year or grade in math and 
science (it does in other subjects), the United States is one of the few nations that does 
consistently perform above the international average. For the most part, the top-ranked group of 
nations has few constant members, particularly among non- Asian countries. Depending on the 
test (subject and grade level), that top group changes membership, with the exception of 
Singapore, which remained at or near the top and scored significantly higher in nearly all science 
and math tests. For example, the United States is just behind Hungary in eighth-grade science 

® Many policy reports and media accounts of the U.S. ranking on these tests fail to report the statistically significant 
differences in ranking and instead report rank orders that are not significantly different. For example, on fourth 
grade science scores in 2003, the United States is ranked fifth but only two countries, Singapore and Japan, have 



12 




scores. Yet, Hungary had a statistically significant decline in performance between 1999 and 
2003, while the United States had a larger and statistically significant improvement in 
performance over the same period. The U.S. ranks lower than Flemish Belgium in eighth-grade 
math in 2003, but Flemish Belgium is in the bottom-ranked group in eighth-grade science, 
trailing far behind the United States. Indeed, there is little consistency in performance or ranking 
among the leading, non-Asian nations. 

Overall, the science and math leaders are the five Asian countries of Chinese Taipei, Hong Kong 
SAR, Japan, Korea, and Singapore, which often, though not always, outperform the United 
States and all the other countries participating in the testing.^ But even so, the significance of this 
“lead” requires more examination to assess the state of the education system and student 
achievement. We need to look at changes over time, population differences, and the relationship 
between performance on these tests and other outcomes of interest. The TIMSS also follows the 
performance of comparable cohorts within each country from one test period to the next. The 
United States was one of only a few nations in which the comparison cohorts consistently 
maintained or improved its test scores over the years. Only Hong Kong and (Latvian-speaking) 
Latvia ranked with the United States in consistently improving or maintaining test scores over 

o 

time and subject. 



The “world” that the United States trails is a haphazard collection of mostly small nations and 
devoid of consistent leaders with a few exceptions. Rather than concluding that the United States 



statistically significantly higher scores (Hong Kong and England’s scores are not “measurably different from the 
U.S. average”) (Gonzales 2004). 

^ For example, an analysis of G-8 country performance on the TIMSS (Miller et al. 2007) notes that “The 
percentages of fourth-grade students at or above the high achievement benchmark in science ranged from 27 percent 
in Scotland to 49 percent in Japan. In the United States, 45 percent of students reached the high benchmark in 
science The percentages of students meeting the advanced benchmark in science ranged from 5 percent in Scotland 
to 15 percent in England (with Japan at 12 percent). In the United States, 13 percent of fourth-graders reached the 
advanced benchmark. Thus, whereas higher percentages of students in Japan than in the other G-8 countries reached 
each benchmark in mathematics, this was not a consistent finding in science.” This is another illustration of the 
shortcomings of single test/grade/year rankings rather than an analysis of consistent patterns and the need to analyze 
the overall results and patterns rather than selected rankings. 

* The United States placed among Cyprus, England, Hong Kong, Latvia, New Zealand and Slovenia on the fourth 
grade math and science tests and Hong Kong, Israel, Korea, Latvia, Lithuania, and the Philippines on the eighth 
grade tests. Only Hong Kong and Latvia saw improvements on both tests, while other the other countries that 
outranked the United States in a particular year showed declines or no change in scores from one test year to the 
next test year. 
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is behind the world, it would be more accurate to conclude that the test results show the United 
States is not the highest performing nation in any single science or math test, but it is one of a 
very few nations that consistently rank above the international average in tests of academic 
performance. And the United States is one of the few that show consistent improvement over 
time and across grades and subjects. What is more, although science and math are the primary 
focus of policy discussion, in other areas, such as literacy, U.S. scores are consistently above the 
international averages. By excluding those tests from international comparisons, it is implied that 
literacy does not hold the same importance as science and math, usually by reference to science 
and math as the drivers of innovation and economic growth. However, there is no substantial 
evidence to support the assertion that a nation’s average levels of math and science mastery lead 
to a disproportionate share of innovation or economic growth. Moreover, employers report that 
literacy and a competence in a broad range of subjects are essential. 

Finally, and despite the fact that many observers do so, it is not meaningful to draw international 
comparisons for the TIMSS results for high school students (often referred to as the “12th grade” 
test). The test is administered to students identified in each nation to be in their final year of 
secondary education. This is problematic because that final year of secondary school varies 
widely across nations from three years up to almost eight years beyond eighth grade. In other 
words, students completing secondary education in many other nations often have received more 
years of education than U.S. students. And there is a strong correlation between years of 
schooling beyond eighth grade and the test scores of secondary students (Boe and Shin 2005). 
Should we be surprised if students who have received more coursework perform better on the 
same test? Consider that a completed secondary education in many European nations is often 
thought to be equivalent to a freshman year in an American college; hence, the ability of 
Europeans to move toward a three-year bachelor’s degree. The twelfth grade TIMSS may reflect 
differences in sequencing of courses rather than performance levels of students who have studied 
the same subject matter. Consequently, the TIMSS has limited value in comparing the relative 
performance of secondary students because the sampled students do not have equivalent 
secondary education. 
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Global Rankings and the PISA. The second international test is the Programme for 
International Student Assessment (PISA), which focuses on assessing more “real life” or applied 
knowledge than the TIMSS and tests 15-year-olds in the OECD countries and a number of non- 
OECD, partner countries. The PISA was administered in 2000, 2003 and 2006. It differs from 
the TIMSS in a number of important ways. Eirst, PISA attempts to test the overall level of 
knowledge and the ability to apply that knowledge in mathematics, science, and reading rather 
than TIMSS’ s goal of attempting to measure performance based directly on school curriculum. 
Second, the PISA includes all the OECD countries, many of which were not in the various 
TIMSS surveys (for example, countries not in the 1999 wave of TIMSS that are in the PISA 
include Germany, Prance, Switzerland, Austria, Spain, Denmark, Norway, Sweden, and 
Pinland). Third, the sample is age -based rather than grade-based, selecting all 15-year-olds rather 
than students in one grade. PISA, like the TIMSS, is widely cited as a test purporting to evaluate 
the math and science performance of students in each country relative to those of other countries 
and to reflect the effectiveness of each country’s education system. It is often cited in U.S. policy 
reports as supposedly showing U.S. students lagging the performance of most other countries. 
Yet, using the PISA results in this manner, as reflecting the comparative performance of the U.S. 
education system or its students, stretches the PISA far beyond its appropriate or even intended 
use. 

The most common interpretation of the PISA results is that it shows U.S. schools are deficient 
compared with those in other countries. However, the PISA researchers state quite clearly that 
these results may indicate something quite different altogether; based on differences in scores 
between countries: 

It cannot automatically be inferred that the schools or particular parts of the education 
system in the first country are more effective than those in the second. However, one can 
legitimately conclude that the cumulative impact of learning experiences in the first country, 
starting in early childhood and up to the age of 15 and embracing experiences both in school 
and at home, have resulted in higher outcomes in the literacy domains that PISA measures. 

(Adams 2003, 381) 



15 




Further, 



PISA is not an assessment of what young people learned during their previous year at 
school, or even during their secondary school years. It is an indication of the learning 
development that has occurred since birth. 

Improving quality and equity therefore require a long-term view and a broad perspective. 

For some countries, this may mean taking measures to safeguard the healthy development of 
young children, or improving early childhood education. For others, it may mean socio- 
economic reforms that enable families to provide better care for the children. But in many, it 
can mean efforts to increase socio-economic inclusion and improve school offerings. (OECD 
2003b, 195) 

The PISA is, by design, not an evaluation of a country’s school system but rather a reflection of a 
number of factors, many and perhaps most importantly, non-school achievement factors. It is 
notable that this central conclusion by the PISA researchers finds little mention in most policy 
reports that, instead, focus quite narrowly on math and science education. We discuss these 
conclusions and implications for policy below. 



There are, at the same time, a number of other factors about the PISA that preclude the use of 
international rankings as typically reported. We note a few of these issues here, but much more 
detailed critiques have been done by others (e.g., Hull 2007; Prais 2003).^ The U.S. sample 
response rate did not initially meet the required minimum PISA established and thus a 
“replacement sample” was used to supplement the initial sample. However, some experts argue 
this replacement sampling technique was not adequate (Prais 2003) and even by PISA standards, 
the U.S. sample was marginal. Perhaps as a consequence, for the United States, some of the 



^ In addition, the PISA shares many of the same problems as with the TIMSS and, importantly, rankings 
typically are typically reported incorrectly, similar to reporting of the TIMSS rankings. Hull (2007: 9) notes: “...in 
reporting 2000 PISA results, to say that the U.S. 15 year-olds rank 15th out of 27 countries in reading is misleading; 
although 14 countries produced numerical scores that are higher than the United States, only three were significantly 
higher while 19 countries scores were statistically no different. The remaining four countries scored significantly 
below the U.S.” 

*** In total, 262 schools and 5,456 students participated in PISA 2003 in the United States, with an average 
of 21 students per school (35 students were selected in each school). However, the initial response rate was 65 
percent so the testing period was extended from the Spring to the Fall of 2003 and, additionally, “replacement 
schools” were selected to reach the final sample of 262 schools (Lemeke et al. 2007). By adding the replacement 
schools, an 83 percent response rate was achieved as based on the initial selected sample (i.e., the replacement 
schools were added to the initial responding sample, and that number [262] was used to calculate the final, weighted 
response rate). Prais (2003) argues that this method of calculating the final response rate is erroneous and, instead, 
the response rate of the second sample should be averaged with the initial response rate rather than using the 
supplementary responses as though they were part of the initial sampling. In terms of representativeness, one 
analysis found, compared with the U.S. school population, responding schools were more likely to be in the West 
and more likely to have fewer Asian or Pacific Islander students and more black, non-Hispanic students (Miller 
2007). 
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changes between the 2000 and 2003 tests appear to reflect sampling error rather than actual 
changes over time/' According to Prais (2003), different countries used different sampling 
criteria, with some countries excluding special needs students (in one German sample, excluding 
special needs students was found to make a difference of 8 points in the average score). The 
PISA researchers note that using age rather than grade to sample students involves a tradeoff as 
the range of grades tested varied quite significantly by country. However, not all countries used 
age-based sampling though the impact of the different sampling frames is not discussed.'^ 
Differences in some cases are quite large: Japan, for example, has 100 percent of its sampled 
students in the 10th grade, Korea and Norway have over 98 percent in the 10th grade, whereas 
the United States has only 61 percent in the 10th grade and a third of the sampled students in the 
9th grade or lower. The impact of the proportion of the sample in 10th grade is not 
inconsequential: “the difference between students in the two grades implies that one school year 
corresponds to an average of 41 score points on the PISA mathematics scale” (OECD 2004, 60). 
How, then, does one compare the United States’ mean score of 483 with Japan’s mean score of 
534, Korea’s score of 542, or Norway’s score of 495? 

Using a standard test across all countries is a difficult undertaking and, predictably, differences 
in context would be expected to account for some of the differences in test scores. For example, 
the mathematics problems were all done in metric units; it would be reasonable to assume that 
students who live in countries using the metric system might have some advantage over those in 
countries that do not use the metric system as their measurement standard. (Some balancing of 
cultural context was attempted by having questions from research teams in different countries, 
but none were submitted by the U.S. team.) 



** For example, there was a significant decline in the relationship between socioeconomic status (SES) and 
student performance for the United States between 2000 and 2003, showing higher scores for lower SES students. 
The impact of socioeconomic status and academic performance has long been found to be a strong relationship and, 
to date, one that is relatively intractable and changed only through intensive effort over long periods of time. A 
change of the magnitude found between the two tests over such a short time period, three years, seems likely to be a 
result of differences in sample than a reflection of changes in the education system during that time period (OECD 
2004, Table 4.3b). 

“To accommodate countries that desired grade-based results for the purpose of national analyses, PISA 
2003 provided an international option to supplement age-based sampling with grade-based sampling” (OECD 2004, 
320). 
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These limitations of the PISA for drawing certain types of comparisons and conclusions are 

discussed by the PISA researchers. They are quite explicit about what they consider the 

appropriate interpretation of the test results: 

As a result of this population definition, PISA makes statements about the knowledge and 
skills of a group of individuals who were born within a comparable reference period, but 
who may have undergone different educational experiences both within and outside schools. 
In PISA, these knowledge and skills are referred to as the yield of education at an age that is 
common across countries. Depending on countries’ policies on school entry and promotion, 
these students may be distributed over a narrower or a wider range of grades. Furthermore, 
in some countries, students in PISA’s target population are split between different education 
systems, tracks or streams. If a country’s scale scores in reading, scientific or mathematical 
literacy are significantly higher than those in another country, it cannot automatically be 
inferred that the schools or particular parts of the education system in the first country are 
more effective than those in the second. However, one can legitimately conclude that the 
cumulative impact of learning experiences in the first country, starting in early childhood 
and up to the age of 15 and embracing experiences both in school and at home, have resulted 
in higher outcomes in the literacy domains that PISA measures. (OECD 2004, 320; emphasis 
added) 

We now turn to the interpretation and implications of international differences in test scores. 



U.S. Students’ Performance on International Tests. For many of the reasons mentioned 
above, the conclusions about a nation’s “international rank” based on the TIMSS and PISA tests 
are overly simplistic and can be very misleading. At the same time, these tests do provide 
important information about the acquisition of several particular types of knowledge and skills. 
Taking into account the tests’ limitations, a number of researchers, including the PISA and 
TIMSS research teams, have analyzed the test results and implications that can be drawn from 
them. 



Boe and Shin (2005) have made a systematic comparison of six different international tests 
administered between 1991 and 2001 by the International Association for the Evaluation of 
Educational Achievement (lEA) and the Organisation for Economic Co-operation and 
Development (OECD). These include the TIMSS and the PISA, as well as other two other tests 
of reading literacy, math/science, and civic knowledge. They aggregate the primary data for 22 
industrialized nations and consider the statistical significance of the original data when ranking 
nations. They conclude that 

U.S. students have generally performed above average in comparisons with students in 
other industrialized nations. Certainly there is variability in performance, with the U.S. 
scoring above average in reading and civics, average in science, and somewhat below 
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average in mathematics. But even in mathematics at the middle and secondary levels, the 
U.S. did not perform “poorly.” (Boe and Shin 2005, 694) 

Their classification collapses minute testing score differences, and they develop national 
groupings according to whether or not their students test the same as U.S. students, better or 
worse. Figure 3 shows their combined ranking on math, science, reading and civics. Only about 
one fifth of other industrial nations have average scores better than those of the United States, 
while roughly four tenths of the nations have average scores the same as those of the United 
States, and another four tenths score worse. Again, U.S. students, on average, are not the top 
scorers in science (25 percent of other nations did better) or in math (44 percent of other nations 
did better). But U.S. students rank highly on reading (only 13 percent of other nations have 
average scores that rank higher); and U.S. students rank with the highest nations on civics scores 
(no other nation ranks better). In short, the United States scores near the top among large 
industrial nations on three out of four fields of knowledge and, while the average math scores do 
not place the United States above the group average, the report card is not as dismal as it is often 
portrayed. 

Boe and Shin conclude that the global divide is between the western industrialized nations and 
the rising Asian nations on math and science test performance, though not in other areas. When 
comparing U.S. student performance to the G7 Nations, the average U.S. test score is 
comparable to the other western nations, all of which trail Japan in mathematics and science. So 
it is not the case that U.S. students are markedly different from students in major European 
nations, but rather that students in Japan and some other Asian nations, on average, perform 
consistently well on these math and science tests. 



NAEP scores show that only 26 percent of U.S. students are proficient or better in math, but U.S. “proficiency” is 
actually highly ranked in international terms; only about a third of students in the leading group of nations score at a 
“proficient” level. 
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Figure 3. Performance of the U.S. Relative to Other Industrialized Nations, 
Median Results of Four Subjects 



Above US 
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Below US 
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Source: Boe and Shin 2005. 






Interpreting the meaning of differences in scores is discussed by the PISA researchers and others, 
though their analysis is rarely considered by the policymakers and others citing the rankings. 
Other aspects of the test-score differences are not discussed by the PISA researchers. We review 
some of the more important issues here. First, even assuming the test is measuring something 
about educational quality, it is difficult to know what the importance is in the “gaps” between 
nations. The raw scores are converted to normalized scores such that 500 is the mean score and 
each 100 points represents one standard deviation. The actual conversion is a weighted measure 
of different raw score components, so from the information provided, it is difficult to know 
exactly what the “knowledge gap” or performance differences are in terms of actual differences 
in correctly answering the test questions. That is, normalizing the raw scores creates a population 
distribution of scores but does not indicate the extent of actual differences in test results. By 
analogy, the finishing times of a group of Olympic marathon runners could be normalized such 
that the difference between 500 and 600 represented one standard deviation in that population. 
For this population of Olympic athletes, a 100 point difference on this scale might represent a 
difference of a few minutes in the finishing times. By contrast, a 100 point difference in 
normalized scores of the Boston Marathon runners might represent a difference of more than an 
hour in finishing times. Thus, for these two different groups of marathon runners, a 100 point 
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difference in the normali z ed scores represents a gap of 10 minutes in one case and more than an 
hour in another case. If the interest is in knowing the importance or meaning of the gap, 
normalized scores are inadequate. 

Without knowing the magnitude of the actual raw score differences on the PISA, we can use the 
test results to rank countries and populations but not know the importance of differences in 
rankings. Do the score differences represent very small gaps in a group of world-class athletes or 
large differences in a mixed group of outstanding performers and poorly performing laggards? 
Or, do they represent inconsequential differences in a group that, on average, performs quite 
poorly? To illustrate from the six sample questions in the PISA report, table 3 shows the 
percentage of students correctly answering each question for the United States and for the 
combined OECD countries, the “performance gap” averaged across all six questions, and the 
scale scores (see Lemke et al. 2004, 7-12 for the test questions). 



Table 3. Percentage of Students Correctly Answering Each Question 



Question: 


1 


2 


3 


4 


5 


6 


Average 
“gap” US- 


Scale 

Score 
















OECD 




U.S. 


62.7% 


74.6% 


20.2% 


67.8% 


37.2% 


39.8% 




483 


OECD 


68% 


72.9% 


25.4% 


73.9% 


40.3% 


32.2% 


-1.7% 


500 


U.S. “Performance 
Gap” 


-5.3% 


+\. 1 % 


-5.2% 


-6.1% 


-3.1% 


+ 1 . 6 % 







Although we do not know if the raw scores for these six questions are representative of all the 
scores in the entire test, they are suggestive that the scale score differences may indicate rather 
small differences in actual scores. And although the test was constructed with careful attention to 
the standard criteria and methods for ensuring test validity, these questions were not related to 
any outcome in national-level performance economically or in science and technology, nor were 
they related to any individual outcomes in career or job performance. Further, are correct 
answers by two-thirds of test takers an indication of poor performance or high levels of 
performance? From the data available in the PISA or TIMSS, it is not clear what, exactly, is 
indicated by particular response rates. 
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Does the level of panic about lagging U.S. performance, and characterizations of a student 
population falling dramatically behind those in other countries correspond to actual performance 
differences of a few percentage points? Or perhaps more to the point, what, exactly, does a 1.7 
percentage point gap mean? Even using the normalized scores, the gap is only 0.17 of one 
standard deviation. Does this really represent a threat to the nation’s science, engineering, or 
innovation capacity? Is a country with a 62 percent correct response rate versus a 64 percent 
correct response rate at a disadvantage in producing leading-edge technology, pioneering 
science, or delivering efficient services or production? There is no empirical basis for drawing 
such conclusions, so it seems the answer is just assumed. Normalized scores are a useful metric 
for representing a population distribution but they do not necessarily provide any insight into the 
importance of the differences, and seldom is the magnitude of the score differences analyzed. 

The question is not about whether to improve the U.S. education system, which is of course a 
worthy goal, but rather why U.S. performance is lower than other countries, what the 
implications are for “competitiveness,” and what policies would best address the deficiencies. 
For this, we need to further examine the reasons for the differences in test scores between 
nations. 

Many researchers note the role of non-school achievement factors found to be strongly 
associated with educational performance levels. The PISA researchers examine these differences 
within and between nations and find that, for most countries, these factors account for a large 
proportion of performance differences. The impact on test scores of single-parent families, for 
which the U.S. rate is two to three times the rate in the countries that have the highest test scores, 
is strikingly large. “Even when controlling for the influence of other socioeconomic factors, an 
average gap of 18 scale points remains between students from single-parent and other types of 
families. This gap is between 25 and 30 score points in Belgium, Ireland, and the United States” 
(OECD 2004, 167). Analysis of the PISA results and socioeconomic factors shows that in some 
countries schools appear to offset the negative impacts to a much greater extent than in other 
countries. The United States is one of six OECD countries where socioeconomic level has a 
strong impact on student performance, where the average score will be more affected by the size 
of the low socioeconomic population, or, conversely, where schools are least effective in 
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providing an education that improves the test performance of students from low socioeconomic, 
immigrant, and/or single-parent families (OECD 2004, 182). 

The PISA researchers repeatedly state, as quoted above, that one cannot use the test results as 
indicative of differences in the education systems of countries. First, of the overall variation in 
student performance, only 10 percent is due to differences between countries. '"Nine-tenths of the 
student performance variation in PISA is within countries” (OECD 2004, 116; emphasis in the 
original). As noted above, it is the overall impact of achievement related factors that affect 
performance and many of these are nonschool factors. An analysis of these nonschool factors 
shows the United States to be quite different from the countries whose students perform higher 
than those in the United States. 

An important difference between the United States and most of the other nations tested is the 
comparative race/ethnic diversity of the U.S. student body and social conditions. In fact, the 
United States stands quite alone in terms of its diversity as, for example, “Germany and Italy 
were nearly 100% white, and Japan’s [population] nearly 100% Asian [and] Canada’s [minority 
population is predominantly] Asian” (Boe and Shin 2005, 693). Boe and Shin analyze the test 
scores of U.S. students and find that white students handily outscore students in the Western G5 
nations in math and science, albeit they do not do as well as Japanese students. On the other 
hand, U.S. white students (with a percentile rank of 92) handily outscore Japanese students on 
reading (with a percentile rank of 69). 

How different is the United States from the group of nations tested? The NCES addressed this 
issue by showing that the United States is not an outlier on a number of different measures 
(single -parent families, foreign bom, etc.) compared with the entire group of nations (NCES 
2006b). But when we compare the United States with industrialized nations whose students test 
better, it is harder to ignore obvious differences. Single-parent households with children under 
age 17 account for 33 percent of families in the United States (U.S. Census 2006), compared 
with 17 percent in Norway (a country with comparatively high levels of social benefits and 
services) and less than 10 percent in Japan, Singapore, and Korea. Almost all of the population 
of Japan, 99 percent, speaks Japanese as their first language, compared with the 18 percent of the 
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U.S. population that lives in a household in which a language other than English is spoken. 

Along these lines, consider that Norway, one of the top-scoring western nations, has a small 
population of 4.5 million with an immigrant population of just 7 percent, of which 44 percent is 
European (with relatively similar social and cultural conditions and background). Although 
Canada has a foreign-born population of 18 percent compared with 1 1 percent of the U.S. 
population, Canada has a much more restrictive immigration policy, effectively limiting 
immigration to high-skilled workers, those establishing a business, and family members of those 
already in Canada. (Canada is one of the few countries in which natives do not significantly 
outperform immigrants [OECD 2004, 167].) 

The United States has a large population and the most diverse demographics of any 
industrialized nation. It is unclear whether using average test scores provide any meaningful 
indication of education or potential economic performance of the United States. What does one 
infer from comparing the average test score in a nation of over 300 million with that of a nation 
of 4.5 million (Singapore) or using educational performance as an indicator of economic 
performance? We would expect India’s 39 percent illiteracy rate and its secondary school 
enrollment rate of less than 50 percent (World Bank 2007) to make it an inconsequential global 
power. Of course, that is not the case because rather than average performance it is the small 
percentage of high performers in a nation of 1 billion that is the more important indicator of its 
relative science and engineering strength. The use of average rates across a diverse group of 
nations and diverse populations is of limited use in drawing conclusions about global standing 
economically or educationally. 

One could argue that it is the diversity and openness of the United States that both contribute to 
its high economic performance and its lower average educational performance. The benefits that 
diversity brings America are accompanied by the challenges of sizeable minority and immigrant 
populations that do not perform as well as the white majority. The often broad chasm that divides 
the educational performance of majority and minority students is partly due to differences in 
such characteristics as language and family formation, but also has much to do with differences 
in the quality of educational systems and household income. New immigrants often come with a 
weak education from their home country and have the highest school dropout rates once in the 
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United States. The test results indicate that, rather than a policy focus on average science and 
math scores, there is an urgent need for targeted educational improvement to serve low- 
performing populations, such as recent immigrants and some minorities. Moreover, the full 
analyses in the PISA test point rather strongly to nonschool achievement factors accounting for 
the variation in test performance, though they also indicate the potential for schools to do a better 
job at mitigating the educational impact of those differences. We point out simply that not only 
do U.S. students on average perform better internationally than reported in a myriad of policy 
papers, but as Boe and Shin demonstrate, the majority of U.S. students (white students) actually 
rank near the very top on international tests. Achievement is known to vary significantly by 
socioeconomic class and race. Understanding the demographic variation in education 
performance is important when drawing conclusions and policy recommendations, as we address 
below. 

Thus, our reading of domestic and international trends suggests that that U.S. schools show some 
steady improvement in math and science, and that the United States is not at any particular 
disadvantage in comparison to most other nations. This, of course, is far different from a claim 
that the school system is performing well or that America’s minorities are being well served, 
where, in both cases, there clearly are problems. But it casts a different light on the meaning of 
average test scores and the supply of S&E qualified pool of graduates, which is large and ranks 
among the best internationally. 
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The Pipeline from High School to Work: Inadequate Supply? 



Having completed the high sehool portion of the pipeline, the ehallenge at the eollege level is to 
attraet students to enter an S&E major and to retain those who enter as S&E majors through 
graduation. In the following seetions, we traee high sehool graduates who go on to eollege and, 
having surmised from the foregoing that large numbers are S&E eompetent, ask whether the 
evidence supports widespread assertions that there is a deelining interest in S&E edueation. Then 
having traeed the pipeline as far as eollege, we follow it further to graduation. We foeus here on 
the aggregate level and all S&E degrees in order to evaluate the broad based assertion that all 
S&E output is in deeline.^"^ 

Undergraduates Show Stable Interest in S&E Majors 

The first elaim about eollege students is that fewer are interested in pursuing an S&E degree. 
While the basis for the elaim is often vague, we assume it refers to the possibility that there is a 
deelining level of interest in pursuing an S&E degree and/or that, of those who do so, a deeline in 
the numbers who eomplete an S&E eourse of study. Despite these eommon assertions, the data 
do not suggest a notable deeline in student interest in S&E eollege edueation. 

A survey of the entering freshman elass asks about their college interests and finds remarkable 
eontinuity in students’ desires to pursue an S&E major. In 1983, about 35 pereent of all entering 
eollege freshman expressed an interest in pursuing an S&E major and, two deeades later, in 
2004, 33 pereent of the entering elass was interested in an S&E major (NSE 2006). Over these 
two deeades, there were some ehanges in intentions by demographie group. The pereentage of 
freshmen males intending to pursue S&E majors deereased from 46 to 41 pereent and so, even 
though a eonstant 25 pereent of female freshman intended to pursue S&E, women inereased 
from 38 to 45 pereent of the entire freshmen population pursuing an S&E major. In faet, the 

We are aware that college patterns differ at the undergraduate and graduate levels, and the college requirements 
differ for different industries. For example, engineering predominantly leads to a bachelor’s degree, whereas the life 
sciences lead to both master’s and doctoral degrees. The university and the research labor market are supplied 
largely by workers with graduate degrees. Also, each labor market and supply chain functions quite differently with 
different demographics and pathways. 

European studies make similar claims about the level of S&E interest in OECD countries (OECD 2003, 2006). 
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interests of most groups changed little except, contrary to popular perception, the percentage of 
Asian American males intending to study S&E declined from 53 to 46 percent. Of course, what 
freshmen intend to do and what they actually do may differ. 

Stable Rates and Increased Numbers of S&E Graduates 

There appears to be rough equivalence in the proportion of students intending to pursue an S&E 
degree and the proportion of domestic students who obtain an S&E degree. Moreover, the 
proportion of all bachelor’s degrees awarded in S&E has been relatively stable over time, as has 
the proportion of freshman in an S&E major. Eigure 4 shows that not quite a third of American 
undergraduates finish an S&E degree, a proportion that has been fairly consistent for over two 
decades. These data are restricted to citizens and permanent residents, whom we refer to as 
“domestic” students, so they are not influenced by the growing number of foreign students 
enrolled in U.S. institutions. 

The story for S&E master’s and doctoral degrees is also primarily one of consistency in the 
proportion of all degrees awarded over the past two decades. About one-fifth of all master’s 
degree graduates obtained their degrees in an S&E field. There was an early 1990s peak but of 
just 1 to 3 percentage points over the 2002 level (the most recent year of available data). The 
master’s degree is more typically awarded to students pursuing professional jobs in business or 
technical jobs in non-S&E fields, because many S&E jobs, such as engineer, and most IT 
occupations do not require a graduate degree. At the same time, the doctoral degree is needed for 
more of the advanced S&E jobs and is not required in most other fields. Not quite two-thirds of 
all doctoral degrees are awarded in S&E; the percentage of all doctoral degrees awarded in S&E 
increased from 56 percent in 1977 to 60 percent in 2002. Most of that increase occurred from the 
later 1970s through the early 1990s, and the proportion has remained more or less stable since 
then. 
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Figure 4. Percent of All Degrees that are S&E Degrees Earned by 
Citizens and Permanent Residents 
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At the same time, the number of students completing an S&E degree has increased. Because the 
proportion of students pursuing an S&E degree has remained stable while the number (or 
proportion) of students pursuing a college degree has increased, more graduates earn an S&E 
degree. Eigure 5 shows that the number of S&E bachelor’s degrees earned by citizens and 
permanent residents has continued to grow smartly at the same time there are growing concerns 
that fewer Americans are pursing S&E degrees. The number of master’s degrees grew steeply 
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through 1995 and has been more or less stable since then. However, the number of doctorates 
earned by citizens and permanent residents appears to have declined a little from the early 1990s 
through 2002; the number of doctorates grew to a peak of about 32,000 in 1995 only to decline 
to about 27,500 in 2002. More recent data just for doctorates suggest supplies of the numbers of 
citizens and permanent have stabilized at around 28,000 through 2004.^^ 

From the start of the late 1970s through today, a fairly stable share of college freshman report 
intending to pursue an S&E degree and a fairly stable proportion graduate with S&E majors. 
There has been no decline in interest by these measures. At the same time, there has been growth 
in the number of undergraduates completing their S&E studies, while the number of S&E 
graduates remains high by historical standards. As to the future, the growth of the college-age 
U.S. population is slowing and growth in the number of youth of college age is slowing as well. 
Yet, the proportion of students pursuing a college degree has been increasing and that may 
somewhat offset the declining population base. Today, about 35 percent of young whites and 
about 55 percent of Asians complete a four-year college education, though only about 20 percent 
of young blacks and about 10 percent of young Hispanics complete college (U.S. Census Bureau 
2007, table 2). How then can we square the argument that there has been, and will be, 
widespread shortages of workers to supply the S&E labor market? 

Students and Jobs: A Missing Link in the Debate? 

The critics assert, or implicitly assume, that the educational pipeline provides an inadequate 
supply of domestic students to work in the S&E labor force. Consequently, the rationale for 
increasing the quantity and quality of K-12 science and math is to increase the flow of S&E 
qualified students into the college segment of the pipeline, which, in turn, is assumed to produce 
more S&E workers. But what if the number of graduates has, in fact, exceeded the number of 
S&E jobs? 



The decline in S&E doctoral graduates was due to fewer foreign-born (permanent) residents earning doctorate 
degrees while the number of citizens changed not at all. The more recent increase in S&E doctoral graduates, an all 
time high in 2005, has been driven by stable citizen numbers and an increase in the graduation of temporary foreign 
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From 1993 to 2002, U.S. colleges produced on average about 380,000 S&E bachelor’s degree 
graduates, over 70,000 master’s degree graduates, and nearly 20,000 doctoral graduates. Is that 
enough? The answer is not straightforward. We need to know what the employment demand is, 
whether the overall supply of graduates interested in entering S&E employment is equal to or 
greater than the number of openings (demand), and whether individuals not entering S&E 
employment are pursuing other careers because they are not interested in an S&E career, or 
could not find a job, or are not qualified for the S&E jobs that are available. 

Are There Enough S&E Graduates? To begin, we would like to know whether the production 
of domestic S&E college students is anywhere near the apparent demand for S&E workers. We 
want to get some idea of how many graduates there are relative to S&E jobs (occupations) and so 
we engage here in a rough and ready exercise. The purpose is simply to get an idea of “order of 
magnitude” and not to engage in false precision. The overall S&E workforce totals about 4.8 
million, which is less than a third of the 15.7 million workers who hold at least one S&E degree. 
The S&E employment of S&E graduates is also a fairly consistent one-third of S&E graduates 
each year. Past employment growth follows this same pattern. Erom 1985 to 2000, the U.S. 
graduated about 435,000 S&E students annually with bachelor’s, master’s, and doctoral 
degrees — that total includes only domestic citizens and permanent residents (about 72 percent of 
S&E workers hold bachelor’s, 20 percent a master’s, and 7 percent a doctorate degree). Over the 
same time period, net change in S&E occupational employment ran about 150,000 annually, 
such that the average ratio of all S&E graduates to net employment ehange was about three to 
one. Of course, net employment growth is not a direct measure of employment demand or total 
job openings, since net growth does not include replacement for retirements or occupational 
quits, nor do these aggregate numbers indicate the types of workers sought (education level. 



students. Over 80 percent of temporary foreign students earn doctorates in an S&E field of study. See NSF, “S&E 
Doctorates Hit All-time High in 2005” [NSE 07-301] at http://www.nsf.gov/statistics/doctorates/. 

In this exercise, we are examining the S&E occupational employment of S&E graduates while 
recognizing that S&E education into S&E employment is only one labor market linkage: an S&E education could be 
valuable in non-S&E occupations, such as a patent lawyer, and many S&E jobs are helped by non-S&E graduates 
(e.g., as much as 40 percent of the IT workforce does not have a four-year degree). We will be conducting more 
detailed analyses of the education to employment flows, but here we wish to provide some broad indicators of 
supply and demand. In addition, typically cited are employer complaints about supposed shortages for S&E jobs, not 
for S&E graduates in other jobs. 

** Calculations made by the authors based on data on graduates and S&E employment for every second year from 
1985 through 2000; and the ratio is based on three-year moving averages of net employment growth. 
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experience, etc). Moreover, it does not address future changes in supply or demand. But it 
certainly is suggestive that plenty of S&E students have been graduating relative to employment 
growth in S&E occupations.^^ 



Leaks in the Pipeline from School to Work. Naturally, not all S&E graduates will enter an 
S&E job, whether because of a change in interest, because their qualifications are not adequate, 
or because they never intended to enter an S&E career in the first place. However, there is a 
surprisingly low rate of S&E retention (high attrition) for the 1993 to 2001 cohorts of S&E 
graduates. One to two years after graduation, 20 percent of S&E bachelor’s are in school but not 
in S&E studies, while another 45 percent are working but in non-S&E employment (total 
attrition of 65 percent). One to two years after graduation, 7 percent of S&E master’s graduates 
are enrolled in school but not in S&E studies, while another 31 percent are working but in non- 
S&E employment (total attrition of 38 percent; NSE, 2006: Table 3). 



Does poor S&E education lead to higher attrition from the S&E pathway? One could surmise 
that poor high school preparation can lead college freshmen to drop out of S&E college majors. 
But research to date finds that once students enter an S&E pathway, prior science and math 
preparation or performance does not appear to be a strong causal factor of attrition. Rather, 
studies identify the quality of college education, be it inadequate teaching resources or poor 
pedagogy, as an important factor leading to attrition among S&E majors during their college 
career. Indeed, the literature on the retention of college-enrolled S&E majors almost uniformly 
identifies the causal factors of attrition as the pedagogical shortfalls of S&E classes rather than 
student performance. Yet, college level outcomes, as discussed above, do not show any notable 
changes in the past decades. And, for example, the need for remediation at the college level has 



This simple calculation appears not to square with a comparison of the annualized growth rate of S&E graduates 
and jobs from 1980 to 2000. That calculation finds that the annual growth rate of S&E graduates at all degree levels 
is about one third that of S&E employment growth (1.5 versus 4.2 percent annually). But the rate of growth 
argument is somewhat misleading insofar as the slower growth rate of S&E graduates is, as noted here, a far larger 
number than the smaller but more rapidly growing number of S&E jobs. At first blush, one might assume these 
sizable differences in growth rates bode poorly for the future, but projections at these rates of growth show that the 
number of graduates and jobs does not converge for about 20 years (see Science and Engineering Indicators, 
Appendix Table 3-2, http://www.nsf.gov/statistics/seind06/pdf_v2.htm). 

A number of studies of attrition of college science and engineering majors find students citing the quality of 
instruction, the “culture” of the discipline, and other curricular issues. One of the most in-depth studies finds that 
these factors are more important than the student’s skill or aptitude (Seymour and Hewitt 1997). 
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not increased notably from the 1980s to the present day, albeit about one-tenth of college 
freshman require science and one-fifth require some math remediation (NSF 2006, table 2). 



Still, students with low achievement may lead to low postgraduate transition and retention rates, 
which could be consistent with the findings of S&E education suffering from poor pedagogy, 
inadequate educational systems, or simply a surfeit of low performers who somehow make it into 
S&E fields of study. Table 3 examines the possibility that large numbers of S&E graduates have 
poor grades and, despite having a degree in hand, are not adequately prepared to either continue 
further S&E studies or take S&E employment. It shows that slightly higher performing students 



Table 3. Employment and Education Status of S&E Degree Recipients by Degree and GPA 



1995 and 2001 


Degree level & GPA 


Graduates 




% In S&E 

(Employed or Continuing in S&E 
Major in School) 


% Employed In S&E 
(of those employed) 




1995 2001 


1995 


2001 


1995 


2001 


Bachelor’s 


698,200 


758,300 


28.8 


33.4 


71.2 


66.5 


3.75-4.0 


83,400 


116,900 


36.2 


35.7 


63.8 


64.4 


2.75-3.74 


524,300 


566,100 


28.3 


33.6 


71.8 


66.3 


Less than 2.75 


89,400 


74,400 


25.4 


28.6 


74.8 


71.4 


Master’s 


146,300 


160,100 


62.1 


62.8 


37.9 


37.1 


3.75-4.0 


33,600 


41,800 


68.4 


66.2 


31.7 


33.8 


2.75-3.74 


100,600 


109,200 


60.6 


62.4 


39.5 


37.7 


Less than 2.75 


11,300 


8,300 


55.3 


51 


44.7 


49 



SOURCE: Adapted from National Science Foundation, Division of Science Resources Statistics, National Survey of Recent 
College Graduates, 1995 and 2001 , special tabulations. Table 2-9, 2003. Science & Engineering Indicators - 2004. Due to 
rounding percents may not add to one hundred 



continue on an S&E pathway after graduation, but there is no dramatic change between 1995 and 
2001. Only about a quarter of S&E bachelor’s students with less than a 2.75 GPA stay on an 
S&E trajectory, while about a third of those with better GPAs stay the course. Similarly, about 
half of S&E master’s students with less than a 2.75 GPA stay on an S&E trajectory, while nearly 
two-thirds of those with better GPAs stay the course. However, for those who enter the job 
market at each stage, a greater proportion of low-GPA graduates find S&E employment than do 
high-GPA students. The lower rate of job entry by high GPA graduates is due, in part, to higher 
GPA students continuing their education rather than entering the job market. However, the lower 
rate may also be the result of higher GPA students entering other, non-S&E careers (a point we 
are examining in current research). A low GPA is apparently not a bar to finding S&E 
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employment. Furthermore, these general patterns have changed remarkably little between 1995 
and 2001 and they indicate high rates of attrition regardless of preparation. 

Is an S&E Education Used in Non-S&E Jobs? Still, an S&E education may be used even 
outside of supposed S&E pathways because many non-S&E occupations may demand a 
substantial amount of S&E knowledge. Some graduates do S&E related work in a job that is 
outside the formal S&E classification, such as a patent lawyer. An S&E education may 
contribute to the jobs for yet other S&E graduates who are not in an S&E occupation, such as a 
manager in a technology firm. Thus, some portion of the non-S&E graduates directly use their 
S&E education in the labor market. The perceived relevance of an S&E education to job 
requirements has been examined for S&E graduates in both S&E jobs and non-S&E jobs. 

Eor S&E job holders there is, curiously, a rather imperfect fit between their education and how 
they perceive the relevance of that education to what they do on the job. A survey of all S&E 
graduates finds that only about 40 percent of bachelor’s degree holders report that their job 
requires skills that are “closely related” to their college major, a share that increases to 61 and 69 
percent for master’s and doctorate degree holders (NSE 2006, tables 3-7). Considering that these 
workers are employed in S&E classified occupations, these percentages do not appear to be 
strikingly high. 

The perceived relevance of an S&E education for those graduates employed outside of S&E is 
very low. Of all workers whose highest degree was in an S&E field and who are employed in 
“other than an S&E occupation,” less than a quarter report that their education is “closely 
related” to their job. Only a select few engineering degrees confer more transferability, such as 
chemical engineering graduates in a non-S&E occupation, of whom 40 percent report that their 
training is useful for their job. These findings do not support the presumption that S&E education 
is generally “needed” by the labor market. Eew S&E graduates report using their S&E-specific 
education when employed outside of S&E occupations. To be sure, a rigorous S&E education 
may be beneficial, but these findings suggest that job holders do not, in fact, highly prize S&E 
education as a general or even as specific qualifier. At the same time, we note that, in general. 

Authors’ tabulations of the 2003 National Survey of College Graduates. 
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graduates in all fields may not perceive their education as highly related to their occupation, and 
thus additional measures are needed to better understand the specific contribution of an S&E 
education to employment in non-S&E fields. 

In short, the U.S has been graduating more S&E students than there have been S&E jobs; hence, 
there are 15.7 million workers who report at least one degree in an S&E field but 4.8 million 
workers in an S&E occupation. There is, rather obviously, high attrition from school to work, 
and it simply cannot be explained by underachieving S&E graduates failing to qualify for jobs. 

At the same time, many of the S&E graduates outside of a formal S&E job may benefit from 
their training, but the simple indicators used here suggest that such training is not central to their 
current employment. This evidence suggests that the school-to-work attrition is neither due to 
poor educational preparation or, more optimistically, to the failure of formal occupational 
classifications to capture the extent to which S&E training is used in the labor market. Something 
else appears to be going on. 

The S&E Job Market: What is the Nature of the Demand? 

The pathway from high school student to college graduate has a number of transition points that 
are the primary focus of current policy initiatives. The goal of these initiatives is to increase the 
flow into, and retention within the S&E education pipeline. However, the data we have reviewed 
suggest that secondary and higher education systems are providing more than adequate supply 
for industry’s hiring needs. Of course, these are aggregate numbers, so there still could be 
shortages for particular occupations or industries; targeted initiatives to increase the flow of 
underrepresented demographic and income groups are warranted to increase workforce 
opportunity and workforce diversity. But overall, addressing the presumed labor market 
problems through a broad-based focus on the education system seems a misplaced effort. 
Whether increasing the supply of S&E educated workforce entrants would have any significant 
impact on workforce supply (given a graduate pool already 50 percent larger than annual 
openings) is a question that requires a better understanding of the labor market for these 
graduates. Moreover, increasing the education supply with such low yields seems a highly 
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inefficient approach without a better understanding of the factors involved in the transition rates 
at all points along the pathway. 

There is little comprehensive, systematic research on how college students choose an S&E 
career, either on the process or the factors that influence those choices. Standard labor market 
economics theory focuses on the marginal impact of wage rate differentials. Research on career 
counseling is focused on matching interests and occupations, based on the assumption that 
interests are more or less fixed. The science and engineering communities have launched 
education and outreach programs to high school students to increase interest in those fields. And 
some observers focus on the overall appeal of an occupation based on its job quality and content 
of work as important factors influencing its attraction to potential entrants. There is some 
research that sheds light on the role of these different factors in labor supply. 

A few labor market studies, notably by Richard Freeman and colleagues (2004, 2006), have 
focused on the quality of S&E jobs. These studies conclude that the decline in the native S&E 
worker pool may reflect a weakening demand, a comparative decline in S&E wages, and labor 
market signals to students about low relative wages in S&E occupations. Indeed, research finds 
that the real wages in S&E occupations declined over the past two decades and labor market 
indicators suggest little shortage (Espenshade 1999). Some researchers see these demand-side 
market forces causing highly qualified students to pursue other careers. A well-accepted model 
of cyclical patterns of student and worker supply is the cobweb model (Freeman 1976). Research 
finds, in accordance with market mechanisms, that an increase in wages leads to an increase of 
job seekers but, in turn, a large supply of job seekers can depress wages. Declining wages will 
result in reduced student enrollments, although there is a lag in enrollment response. For 
example, research finds that a previous decline in mathematics enrollments through 1996 
corresponded to this cycle (Davis 1997). 

Where’s the Problem? Hiring Difficulties versus Labor Market Shortages 

The assertion that education deficits are directly linked to employer reports of hiring difficulties 
is generally stated without much examination of the evidence. It is important to distinguish 
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between problems an employer may have hiring the people he or she wants and an aetual 
shortage of workers or potential workers. Although there may be a labor market shortage, all the 
evidenee of “shortages” eited in the various poliey reports is entirely individual employer 
aeeounts of problems in hiring. The industries most voeal about labor market shortages and the 
need to import workers may be voieing unrealistie expeetations of experienee more than skills or 
edueation of a new hire, or just eost. In previous research (Lynn and Salzman 2002), we found 
that managers in engineering and technology firms do not claim a shortage of applicants nor do 
they complain of applicants with poor math and science skills or education. They do often note 
difficulty in finding workers with sufficient experience, specific technical skills, or a sufficient 
number of “brilliant” workers in the pool. The complaint, quite often, appears to be one of 
unrealistic expectations, as unwittingly illustrated in a recent BusinessWeek (2007) article on 
labor shortages: “‘There are certain professions where skills are in such demand that even 
average or below-average people can get hired,’ says Michael Alter, president of SurePayroll 
Inc.” Other than frustration at not having an applicant pool at the tail-end of the skill distribution, 
the skills deficit most likely to be mentioned are the “soft skills” of communication and the ease 
of working across organizational, cultural, and disciplinary boundaries (Lynn and Salzman 2002; 
Salzman 2000). Science and engineering firms most often complain about schools failing to 
provide students with the non-technical skills needed in today’s firm. 

It is also worth noting that, more generally, employers do not complain about the math and 
science skills of employees hired for professional positions. In a study of engineering skills, 
technical qualifications were not identified as a concern by managers. Employers’ complaints 
about math skills typically involve examples of retail workers who can’t count change or clerical 
applicants who lack basic literacy. And even for these levels the need is for a broad array of 
academic, social, and communication skills (Murnane and Levy 1996). 



Employers may complain of difficulties in hiring experienced workers with specific skills, such as JAVA 
programmers with 10 years experience, but these “shortages” are not the result of insufficiencies in the education 
system. 

In our interviews with engineering managers (Lynn and Salzman 2002), rarely, if ever, do they say they are 
unable to find graduates with the requisite technical skills but rather the “shortage” is of engineers with 
communication, management, interpersonal and other soft skills. 
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Traditional views of the labor market foeus on wage rates and vaeaneies — are there positions 
unfilled and are wages rising. Employers foeus on the diffieulty of hiring the eandidate they want 
whieh is a eombination of people willing to aeeept the wages offered, having the skills and other 
attributes desired, and willing to work in their firm. Market eeonomies suggests ehanges in wage 
rates should be suffieient to balanee supply and demand. It is not elear that inereasing the pool of 
potential workers by expanding the number with the requisite edueation would ease hiring 
diffieulties through sheer quantity. Moreover, it is an approaeh with low effieieney sinee there is 
already a large pool with suffieient human eapital who ehoose not to enter, or subsequently leave 
after entering an S&E job, representing substantial private and publie edueational investment that 
is not being utilized. 

If, as we argue, there is suffieient potential workforee but firms are unable to eonvert the 
potential workforee into aetual S&E workers, then it is important to examine other faetors that 
influenee eareer deeisions and hiring diffieulties. In addition to wages, there is also the impaet of 
pereeived eareer opportunities and uneertainty. The eurrent heated debate about offshoring of 
engineering and other high-skill work should be expeeted to have an impaet on students’ eareer 
ehoiees. Although some analyses find relatively small numbers of jobs lost to offshoring, the 
perception oboui future opportunity is likely to be as, or more important a determinant in a 
student’s assessment of future opportunities than tallies of eurrent jobs available. These 
pereeptions are not just the result of inflamed media eommentators; even the business 
eommunity appears to be undeeided about the future eourse of its job loeation deeisions. Eor 
example, in a bid to inerease visa eaps, a number of high-teeh CEOs diseussed the demand their 
eompanies had for U.S. -based seienee and engineering workers to a Wall Street Journal reporter 
in June, 2006, 

Mr. McNealy says Sun does 75% to 80% of its research and development in the U.S. 

Craig Barrett, chairman of Intel Corp., says his company also employs most of its 
researchers in the U.S. and wants to keep it that way. The reasons? . . . “If engineering is 
happening here in the U.S., I think my children will have a richer work environment.” 

. . .Moreover, Messrs. Barrett, McNealy, and Gates all say that it helps to have a 
concentration of researchers in the same place, where they can interact over the water 
cooler and at the baseball game, as well as on the computer screen. (Wall Street Journal 
2006) 
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However, graduating college students might have been influenced by an announcement Sun 
made to Wall Street analysts in May of 2005: 

Sun Microsystems Inc. has chosen four of its facilities around the world to take the place 
of its Silicon Valley office as the research and development hub. . . . “We are over-invested 
in high-cost geographies like the U.S., and underinvested in low-cost geographies like 
India,” Stephen Pelletier, the company's senior vice president of global engineering, told 
reporters in Bangalore. Pelletier said the company will not lay off programmers in the 
U.S. — but won't hire many, either.. . . The company has reduced its staff to about 30,000, 
from roughly 43,000 four years ago. (Associated Press 2005; emphasis added) 

One can imagine that companies who are offshoring would have hiring problems even with an 
adequate labor market supply in the United States. Similarly, IT executives calling for greatly 
increasing, or even completely removing, numerical caps on foreign worker visas (e.g. the H-IB) 
may be sending strong signals to students and current workers about diminished career 
opportunities. Human capital is a long-term investment and potential S&E students read all the 
tea leaves before investing. We have conducted interviews with current managers and engineers 
who believe that there is little future in entry-level engineering jobs in many industries, and IT in 
particular. Not only will it be difficult to fill mid-level and higher-level positions from an 
inexperienced workforce that never had an entry-level position, but several future generations of 
workers, currently in school, are developing their work interests and career aspirations based on 
their perceptions about the future state of labor markets. A range of public policies, such as 
immigration policy and corporate practices, such as offshoring R&D, affects the current 
workforce and future generations as well. 

There is also some evidence that the opportunity in the engineering field, the content of the work, 
and the overall working conditions are less appealing today than in the past. From our current 
study of engineering, we often heard engineers and managers noting the lack of motivating 
science and engineering “problems” or challenges, like those of the early days of IT, and the lack 
of national purpose that was evident during the heyday of the space program. Engineers and 
managers interviewed also pointed to changes in both the “substance” and process of 
engineering. Projects are larger, team efforts, and require more coordination and management 
(whether because of outsourcing, systems integration, or increased scale of the technology, such 
as large enterprise resource planning systems). Developing and building many artifacts may be 
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more routinized and less ehallenging or interesting than before. As one colleague expressed it, 
“How many ‘real’ engineers does it take to build a bridge?” These are attributes of both the 
intrinsic interest of the field and the cultural milieu, or Zeitgeist, of science and engineering. 
Although these factors are difficult to measure, they were noted by interviewees as often as 
diminished job prospects in explaining why they would not enter the field today. 

Some S&E graduates simply leave the field because they lose interest in the application of their 
training or, more prosaically yet, they find that the labor market pays more for them to take other 
jobs (e.g.. Freeman 2006). It is thus important to examine the full spectrum of labor market 
signals that can influence student and worker career choices. There are multiple dimensions of 
career choice and identify factors that influence these outcomes. 

Finally, it would be important to understand the different S&E labor markets by industry, 
occupation, geography, and demographic. The labor market studies examine market conditions 
that may influence career choice in aggregate. Fess often do these studies examine choices by 
different demographic groups on entering specific S&E occupations or industries. For example, 
some S&E occupations appear to attract large numbers of traditional S&E students — U.S. native 
white males — but in others females outnumber males, and other occupations are 
disproportionately filled by immigrants. In addition to documenting overall labor market trends, 
it is important to understand specific industry dynamics. The IT industry labor market may be 
different from that of biotechnology or mechanical engineering (e.g., 40 percent of the IT 
workforce does not have a four-year degree; biotechnology has one of the largest concentrations 
of Ph.D.’s in industry; engineering is predominantly a bachelor’s degreed workforce). Although 
the labor market analyses examine changes in relative wages for S&E jobs and non-S&E jobs 
with similar education requirements (e.g., other professional jobs), they have not so far 
determined what affects the industry and occupation decisions of today’s young people who 
could potentially enter S&E careers. 

Michael Horrigan an economist at Bureau of Labor Statistics suggests that between the advances in knowledge for 
many engineering undertakings and technology shifts, say in using more engineering software, the role of 
engineering has likely changed and it may be that fewer jobs involve the engineering challenge of yesteryear 
(Personal communication, January 13, 2006). In our studies of engineering, we find that outsourcing and offshoring 
lead to new engineering management layers and engineers comment that they now manage engineering project 
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Conclusion and Policy Discussion 



Current policy is driven by the twin perceptions of a labor market shortage of scientists and 
engineers and of a pool of qualified students that is small in number and declining in quality. 
Math and science education are viewed as the primary policy levers to increase labor market 
supply, supplemented by increased immigration. But those policy proposals that call for more 
math and science education, aimed at increasing the number of scientists and engineers, do not 
square with the educational performance and employment data that we have reviewed. Our 
review of the data finds not only little evidence to support those positions and, in fact, the 
available evidence indicates an ample supply of students whose preparation and performance has 
been increasing over the past decades. We are concerned that the consensus prescriptions are 
based on some misperceptions about efficient strategies for economic and social prosperity. 

It is difficult to conclude that the major economic “threats” to the United States are related to the 
performance levels of U.S. students as compared to students in other countries. Our major 
economic competitors, particularly emerging nation behemoths, are not among top test scoring 
nations. In fact, a sizeable portion of U.S. students perform at the top of the scale and graduate in 
substantial numbers. The logic of the education crisis reports fails on several of their key points. 
These reports focus on countries that score higher than the United States, primarily just on math, 
and then conclude these countries pose a “threat” to the U.S. economy. Should U.S. policy be 
driven by test score performance of students in Flemish Belgium, Latvian- speaking Latvia, or 
even Singapore, with 4.5 million people and a workforce of 2.4 million (one-sixtieth the size of 
the U.S. workforce)? How will these countries find the capital and the numbers of workers 
needed to “steal” any major portion of a U.S. industry? 

Perhaps one should, instead, look at the countries that are “competing” with the United States 
and examine the ways in which they are doing so and the ways in which the United States is, in 
comparison, deficient. As noted, nearly all the major global powers are not even on the list of 



rather than engage in “real” engineering. Others have commented that engineering is less central to “innovation” or 
at least product development than design, marketing, and other areas. 
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leading test scoring countries. One of the countries that is a leading technology force, Singapore, 
is trying to emulate U.S. innovation and creativity and de-emphasize strict math and science test 
performance. Will better math and science performance increase the number of software 
programming jobs remaining in the United States? This seems unlikely until the Beijing or 
Bangalore wage is on parity with U.S. wages. Will better math and science education improve 
innovation? There is little empirical support for the argument that more math and science 
education will produce more scientists and engineers which, in turn, will increase the level of 
innovation. Nor does research establish that better test scores will improve the innovation 
performance of scientists and engineers who enter the workforce. Some even argue that science 
and engineering are less important economic drivers than they were in the past (Hill, 2007). 

There is an unexamined assumption that the United States is best served by the goal to be first in 
the world in science and math test scores. This confuses means and ends. First, the means to 
improving education is probably not through a narrow focus on math and science. The math and 
science deficits are not among the populations who are well-educated but, rather, the research 
suggests, those with non-school factors that hinder academic performance. As Boe and Shin 
argue, “The U.S. is not ‘first in the industrialized world’ in minimizing the percentage of its 
population living in poverty. . .. So why should anyone expect the U.S. to be first in the world in 
educational achievement? There is, after all, abundant evidence that these types of social 
indicators are strongly associated with educational achievement” (2005, 694). A focus on the 
average test scores obscures the distribution of performance and its root causes. Policy reports 
that focus on the performance of the upper end of the distribution (e.g., more AP classes and 
other advanced math and science coursework) assume that improvements here will increase S&E 
workforce size. 



Perhaps tellingly, Singapore’s recent “competitiveness” policy focuses on creativity and developing a 
more broad-based education. Its other central workforce development strategy is filling in the island’s surrounding 
ocean so that it can expand its total population by 2.5 million. In contrast to the policy focus of U.S. competitiveness 
committees calling for the U.S. to emulate Singapore’s math and science education programs, Singapore 
government proclamations include statements such as: “Creative Industries have been recognised by the Economic 
Review Committee in 2003 as a new and promising area to grow and play a vital role in differentiating Singapore's 
value proposition.” (Creative Industries Singapore; http://www.creativeindustries.sg/public/strategy/) “The new 
architects of the global economic landscape are those who apply their imagination, creativity and knowledge to 
generate new ideas and create new value. Multi-dimensional creativity — artistic creativity, business 
entrepreneurship and technological innovation — will be the new currency of success.” 
http://www.mica.gov.sg/mica_business/b_creative.htmI 
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But low averages and the pool of future high performers must be addressed by improving the 
performance of particular disadvantaged groups that face barriers to education that are not 
addressed by expanding or even improving math and science education. If improvement in 
national test score performance is the goal, greater improvements are more likely to come from 
improving the test scores of those in the lower end of the distribution. Although there are 
initiatives to address these problems, they need to be stepped up in “competitiveness” policy 
because these groups account for much of the international “performance gap.” Efficiency alone 
would dictate a focus on these groups as the most effective policy to increase U.S. performance. 

Another policy assumption is that test performance and more scientists and engineers will 
improve economic performance. The TIMSS analysis suggests curriculum changes would 
increase test scores by narrowing the curriculum and focusing on particular math and science 
components of the curriculum. Yet, it seems a different question should be asked: What are the 
factors that have led to the consistent high performance of the U.S. economy? Which factors 
have provided the U.S. economy more consistent growth than countries that more consistently 
score high on international tests? And what kind of workforce is likely to improve prospects of 
the United States in the future? 

In a provocative article, Ramirez et al. (2006) argue that policies to increase economic growth by 
improving aggregate student achievement are “not based on research evidence” (2006:1). They 
develop a number of regression models to test the relationships between educational performance 
and economic growth for nearly 40 countries over the period 1970 through 2000. They find that 
student achievement levels in math and science “has no effect on tertiary enrollment in science 
and engineering” (p. 17), and a tenuous relationship between educational achievement levels and 
national economic performance. They argue that the four Asian countries of South Korea, Japan, 
Singapore and Hong Kong are outliers and that their high academic achievement is endogenous. 
Without those countries, there is no cross-national relationship between level of education and 
national economic performance. This is also consistent with the TIMSS researchers finding of no 
relationship between TIMSS ranking and GNP. Instead, Ramirez et al. argue, “student 
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achievement is an indicator of national commitments to development rather than a means to this 
development” (p.l5)?^ 

Finally, assessing the claims of labor market shortages is crucial. Purported labor market 
shortages for scientists and engineers are anecdotal and also not supported by the available 
evidence. Little analysis has been conducted of hiring difficulties by firms and the supply of 
workers. A particular employer or industry’s experiences in hiring could be the result of any 
number of factors. The assumption that difficulties in hiring is due just to supply can have 
counterproductive consequences: an increase in supply that leads to high unemployment, 
lowered wages and decline in working conditions will have the long-term effect of weakening 
future supply by discouraging current students. Moreover, by bringing immigrants directly into 
the S&E workforce but without the attachments immigrants develop through longer residency 
and schooling in the United States, there is likely to be greater geographical workforce mobility. 
As the physical infrastructure of emerging nations improves, the location of innovation and R&D 
is likely to follow rather than determine the location of human capital. Investing in domestic 
human capital can provide longer-term benefits to the United States and a collaborative approach 
with those countries will capture the benefits of their human capital development rather than 
trying to absorb it through short-term immigration to address short-term hiring needs (Lynn and 
Salzman 2006, 2007). The characteristics of human capital development and employment are 
qualitatively different from that of prior periods, and we should not fall back on past approaches 
to policy. Instead, evidence-based policy is necessary for developing effective programs for the 
emerging global economy. 



They further suggest that “much of the achievement ‘effect’ is not really causal in character. It may be, rather, that 
nation-states with strong pro-development policies, and with regimes powerful enough to enforce these, produce 
both more economic growth and more disciplined student-achievement levels in fields (e.g., science and 
mathematics) perceived to be especially development related” (p.l5). But even this “Asian exceptionalism” is weak 
when the recent period of slow growth is included in the equations. Moreover, “the apparent achievement effect 
decreases (rather than increases, as we expect) with the increased level of educational enrollments in a country.” 
They find that, to the extent that educational achievement levels are related to low economic growth, it is only 
nations at the bottom third of the educational distribution that have lower performance with no to little difference 
among countries in the upper two-thirds of the educational rankings. 
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Appendix 



Table Al. Employment Status of Recipients of S&E Bachelor's Degrees in 1996/97 and 1997/98, 
by Eield of Degree, Sex, Race/Ethnicity, 1999 


Field of degree, sex, race/ethnicity 


Total 

Number 


% in 
School 


Employed in an 
S&E occupation 


Employed in a 
non-S&E 
occupation 








(% of those not in school fulltime**) 


Computer/math sciences 


69,800 


10% 


49% 


46% 


Male 


46,900 


11% 


54% 


42% 


Female 


22,900 


9% 


40% 


54% 


White, non-Hispanic 


49,200 


10% 


48% 


47% 


Asian/Pacific Islander 


9,200 


S 


S 


S 


Black, non- 

Hispanic 


6,500 


S 


S 


S 


Hispanic 


4,600 


S 


S 


S 












Life and related sciences 


164,000 


33% 


19% 


72% 


Male 


73,000 


34% 


19% 


75% 


Female 


91,100 


32% 


19% 


70% 


White, non- 

Hispanic 


123,300 


31% 


18% 


75% 


Asian/Pacific Islander 


21,200 


46% 


S 


56% 


Black, non- 

Hispanic 


9,000 


34% 


S 


78% 


Hispanic 


9,700 


35% 


S 


65% 












Physical and related sciences 


36,500 


35% 


46% 


50% 


Male 


22,500 


33% 


49% 


47% 


Female 


14,100 


36% 


40% 


54% 


White, non-Hispanic 


29,800 


34% 


46% 


50% 


Asian/Pacific Islander 


2,900 


48% 


S 


S 


Black, non-Hispanic 


2,100 


38% 


38% 


54% 


Hispanic 


1,600 


25% 


S 


58% 












Engineering 


114,600 


13% 


79% 


18% 


Male 


92,000 


12% 


79% 


18% 


Female 


22,600 


16% 


76% 


19% 


White, non-Hispanic 


83,100 


11% 


81% 


16% 


Asian/Pacific Islander 


17,300 


23% 


77% 


19% 


Black, non-Hispanic 


5,800 


12% 


61% 


39% 


Hispanic 


7,900 


8% 


73% 


22% 


S data with weighted values less than 100 or unweighted sample sizes less than 20 are suppressed for reasons of respondent confidentiality 
and/or data reliability 


Notes'. Includes all those who received S&E bachelor’s degrees between July 1996 and June 1998. Figures are rounded to the nearest 
hundred. Details may not add to totals because of rounding. Totals include those of other or unknown race/ethnicity. 


** May not add to 100% because some are out of school and unemployed 


Source: National Science Foundation, Division of Science Resources Statistics, 
National Survey of Recent College Graduates. 
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